Data processing apparatuses, methods, computer programs and computer-readable media

ABSTRACT

First and second pluralities of residual elements useable to reconstruct first and second respective parts of a representation of a signal are obtained. A transformation operation is performed to generate at least one correlation element. The transformation operation involves at least one residual element in the first plurality and at least one residual element in the second plurality. The at least one correlation element is dependent on an extent of correlation between the at least one residual element in the first plurality and the at least one residual element in the second plurality. The transformation operation is performed prior to the at least one correlation element being encoded.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. application Ser. No.16/295,851, filed Mar. 7, 2019, which is a continuation of InternationalApplication No. PCT/GB2017/052632, filed Sep. 8, 2017, which claimspriority to UK Application No. GB1615266.2, fled Sep. 8, 2016, under 35U.S.C. § 119(a). Each of the above-referenced patent applications isincorporated by reference in its entirety.

BACKGROUND OF THE INVENTION Field of the Invention

The present invention relates to data processing apparatuses, methods,computer programs and computer-readable media.

Compression and decompression of signals is an important considerationin many known systems.

Many types of signal, for example video, audio or volumetric signals,may be compressed and encoded for transmission, for example over a datacommunications network. Other signals may be stored in a compressedform, for example on a storage medium such as a Digital Versatile Disc(DVD). When such a signal is decoded, it may be desired to increase alevel of quality of the signal and/or recover as much of the informationcontained in the original signal as possible.

Some known systems exploit scalable encoding techniques. Scalableencoding involves encoding a signal along with information to allow thereconstruction of the signal at different levels of quality, dependingon the capabilities of the decoder and the available bandwidth. However,relatively large amounts of information may need to be stored and/ortransmitted, particularly as the usage of higher quality, higherdefinition video becomes more widespread.

BRIEF SUMMARY OF THE INVENTION

According to a first aspect of the present invention, there is providedapparatus configured to: obtain a first plurality of residual elementsuseable to reconstruct a first part of a first representation of asignal at a relatively high level of quality using a first part of asecond representation of the signal at the relatively high level ofquality, the first part of the second representation being derivablefrom a first set of one or more signal elements in a representation ofthe signal at a relatively low level of quality; obtain a secondplurality of residual elements useable to reconstruct a second,different part of the first representation of the signal using a second,different part of the second representation of the signal, the secondpart of the second representation being derivable from a second,different set of one or more signal elements in the representation ofthe signal at the relatively low level of quality; and perform at leastone transformation operation involving at least one residual element inthe first plurality of residual elements and at least one residualelement in the second plurality of residual elements to generate atleast one correlation element, the at least one correlation elementbeing dependent on an extent of correlation between the at least oneresidual element in the first plurality of residual elements and the atleast one residual element in the second plurality of residual elements,wherein the apparatus is configured to perform the at least onetransformation operation prior to the at least one correlation elementbeing encoded.

According to a second aspect of the present invention, there is providedapparatus configured to: perform at least one transformation operationinvolving at least one correlation element to generate at least oneresidual element in a first plurality of residual elements and at leastone residual element in a second plurality of residual elements, the atleast one correlation element being dependent on an extent ofcorrelation between the at least one residual element in the firstplurality of residual elements and the at least one residual element inthe second plurality of residual elements, the first plurality ofresidual elements being useable to reconstruct a first part of a firstrepresentation of a signal at a relatively high level of quality using afirst pail of a second representation of the signal at the relativelyhigh level of quality, the second plurality of residual elements beinguseable to reconstruct a second, different part of the firstrepresentation of the signal using a second, different part of thesecond representation of the signal; derive the first part of the secondrepresentation from a first set of one or more signal elements in arepresentation of the signal at a relatively low level of quality;derive the second part of the second representation from a second,different set of one or more signal elements in the representation ofthe signal at the relatively low level of quality; and use at least thefirst plurality of residual elements, the second plurality of residualelements, the first part of the second representation and the secondpart of the second representation to generate output data, wherein theapparatus is configured to perform the at least one transformationoperation following the at least one correlation element being decoded.

According to a third aspect of the present invention, there is provideda method comprising: obtaining a first plurality of residual elementsuseable to reconstruct a first part of a first representation of asignal at a relatively high level of quality using a first part of asecond representation of the signal at the relatively high level ofquality, the first part of the second representation being derivablefrom a first set of one or more signal elements in a representation ofthe signal at a relatively low level of quality; obtaining a secondplurality of residual elements useable to reconstruct a second,different part of the first representation of the signal using a second,different part of the second representation of the signal, the secondpart of the second representation being derivable from a second,different set of one or more signal elements in the representation ofthe signal at the relatively low level of quality; and performing atleast one transformation operation involving at least one residualelement in the first plurality of residual elements and at least oneresidual element in the second plurality of residual elements togenerate at least one correlation element, the at least one correlationelement being dependent on an extent of correlation between the at leastone residual element in the first plurality of residual elements and theat least one residual element in the second plurality of residualelements, wherein the at least one transformation operation is performedprior to the at least one correlation element being encoded.

According to a fourth aspect of the present invention, there is provideda method comprising: performing at least one transformation operationinvolving at least one correlation element to generate at least oneresidual element in a first plurality of residual elements and at leastone residual element in a second plurality of residual elements, the atleast one correlation element being dependent on an extent ofcorrelation between the at least one residual element in the firstplurality of residual elements and the at least one residual element inthe second plurality of residual elements, the first plurality ofresidual elements being useable to reconstruct a first part of a firstrepresentation of a signal at a relatively high level of quality using afirst part of a second representation of the signal at the relativelyhigh level of quality, the second plurality of residual elements beinguseable to reconstruct a second, different part of the firstrepresentation of the signal using a second, different part of thesecond representation of the signal; deriving the first part of thesecond representation from a first set of one or more signal elements ina representation of the signal at a relatively low level of quality;deriving the second part of the second representation from a second,different set of one or more signal elements in the representation ofthe signal at the relatively low level of quality; and using at leastthe first plurality of residual elements, the second plurality ofresidual elements, the first part of the second representation and thesecond part of the second representation to generate output data,wherein the at least one transformation operation is performed followingthe at least one correlation element being decoded.

According to a fifth aspect of the present invention, there is provideda computer program comprising instructions which, when executed, causean apparatus to perform a method comprising: obtaining a first pluralityof residual elements useable to reconstruct a first part of a firstrepresentation of a signal at a relatively high level of quality using afirst part of a second representation of the signal at the relativelyhigh level of quality, the first part of the second representation beingderivable from a first set of one or more signal elements in arepresentation of the signal at a relatively low level of quality;obtaining a second plurality of residual elements useable to reconstructa second, different part of the first representation of the signal usinga second, different part of the second representation of the signal, thesecond part of the second representation being derivable from a second,different set of one or more signal elements in the representation ofthe signal at the relatively low level of quality; and performing atleast one transformation operation involving at least one residualelement in the first plurality of residual elements and at least oneresidual element in the second plurality of residual elements togenerate at least one correlation element, the at least one correlationelement being dependent on an extent of correlation between the at leastone residual element in the first plurality of residual elements and theat least one residual element in the second plurality of residualelements, wherein the at least one transformation operation is performedprior to the at least one correlation element being encoded.

According to a sixth aspect of the present invention, there is provideda computer program comprising instructions which, when executed, causean apparatus to perform a method comprising: performing at least onetransformation operation involving at least one correlation element togenerate at least one residual element in a first plurality of residualelements and at least one residual element in a second plurality ofresidual elements, the at least one correlation element being dependenton an extent of correlation between the at least one residual element inthe first plurality of residual elements and the at least one residualelement in the second plurality of residual elements, the firstplurality of residual elements being useable to reconstruct a first partof a first representation of a signal at a relatively high level ofquality using a first part of a second representation of the signal atthe relatively high level of quality, the second plurality of residualelements being useable to reconstruct a second, different part of thefirst representation of the signal using a second, different part of thesecond representation of the signal; deriving the first part of thesecond representation from a first set of one or more signal elements ina representation of the signal at a relatively low level of quality;deriving the second part of the second representation from a second,different set of one or more signal elements in the representation ofthe signal at the relatively low level of quality; and using at leastthe first plurality of residual elements, the second plurality ofresidual elements, the first part of the second representation and thesecond part of the second representation to generate output data,wherein the at least one transformation operation is performed followingthe at least one correlation element being decoded.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 shows a schematic block diagram of an example of a signalprocessing system in accordance with an embodiment of the presentinvention;

FIG. 2 shows a schematic diagram of an example of a signal processingtechnique in accordance with an embodiment of the present invention:

FIG. 3 shows a schematic diagram of another example of a signalprocessing technique in accordance with an embodiment of the presentinvention; and

FIG. 4 shows a schematic block diagram of an example of an apparatus inaccordance with an embodiment of the present invention.

DETAILED DESCRIPTION OF THE INVENTION

Referring to FIG. 1, there is shown an example of a signal processingsystem 100. The signal processing system 100 is used to process signals.Examples of types of signal include, but are not limited to, videosignals, image signals, audio signals, volumetric signals such as thoseused in medical, scientific or holographic imaging, or othermultidimensional signals.

The signal processing system 100 includes a first apparatus 102 and asecond apparatus 104. The first apparatus 102 and second apparatus 104may have a client-server relationship, with the first apparatus 102performing the functions of a server device and the second apparatus 104performing the functions of a client device. The signal processingsystem 100 may include at least one additional apparatus. The firstapparatus 102 and/or second apparatus 104 may comprise one or morecomponents. The components may be implemented in hardware and/orsoftware. The one or more components may be co-located or may be locatedremotely from each other in the signal processing system 100. Examplesof types of apparatus include, but are not limited to, computeriseddevices, routers, workstations, handheld or laptop computers, tablets,mobile devices, games consoles, smart televisions, set-top boxes, etc.

The first apparatus 102 is communicatively coupled to the secondapparatus 104 via a data communications network 106. Examples of thedata communications network 106 include, but are not limited to, theInternet, a Local Area Network (LAN) and a Wide Area Network (WAN). Thefirst and/or second apparatus 102, 104 may have a wired and/or wirelessconnection to the data communications network 106.

The first apparatus 102 comprises an encoder device 108. The encoderdevice 108 is configured to encode signal data. The encoder device 108may perform one or more further functions in addition to encoding signaldata. The encoder device 108 may be embodied in various different ways.For example, the encoder device 108 may be embodied in hardware and/orsoftware.

The second apparatus 104 comprises a decoder device 110. The decoderdevice 110 is configured to decode signal data. The decoder device 110may perform one or more further functions in addition to decoding signaldata. The decoder device 110 may be embodied in various different ways.For example, the decoder device 110 may be embodied in hardware and/orsoftware.

The encoder device 108 encodes signal data and transmits the encodedsignal data to the decoder device 110 via the data communicationsnetwork 106. The decoder device 110 decodes the received, encoded signaldata and generates decoded signal data. The decoder device 110 mayoutput the decoded signal data, or data derived using the decoded signaldata. For example, the decoder device 110 may output such data fordisplay on one or more display devices associated with the secondapparatus 104.

In some examples described herein, the encoder device 108 transmits tothe decoder device 110 a representation of a signal at a given level ofquality and information the decoder device 110 can use to reconstruct arepresentation of the signal at one or more higher levels of quality.

Compared to some known techniques, the examples described herein involvetransmitting a relatively small amount of information to be used forsuch reconstruction. This reduces the amount of data transmitted via thedata communications network 106. The savings may be particularlyrelevant where the signal data corresponds to high quality video data,where the amount of information transmitted in known systems can beespecially high.

Examples described herein facilitate more selective encoding of theinformation the decoder device 110 uses to reconstruct therepresentation of the signal at the one or more higher levels ofquality. In particular, examples described herein convert suchinformation into a form that allows fine-grained control over encodingof individual data elements. This facilitates more efficient andeffective encoding, for example where the encoding takes into accountthe relative importance of the data elements being encoded.

Referring to FIG. 2, there is shown schematically an example of a signalprocessing technique 200. The signal processing technique 200 may beperformed by the first apparatus 102.

The first apparatus 102 obtains input data 202. For example, the firstapparatus 102 may receive the input data 202 from one or more otherentities.

The input data 202 comprises a representation of a signal at arelatively high level of quality. The relatively high level of qualityis high relative to one or more other levels of quality in a tieredhierarchy having multiple different levels of quality. The relativelyhigh level of quality may be the highest level of quality in the tieredhierarchy or may be an intermediate level of quality.

In this example, the input data 202 is arranged as a 4×4 array of signalelements comprising four rows and four columns of signal elements. Theinput data 202 may for example comprise an image and each of the signalelements may represent a pixel comprised in the image.

In this example, the input data 202 comprises four sets of signalelements. A signal element I_(ij) corresponds to the j^(th) signalelement in the i^(th) set of signal elements in the input data 202. Thedifferent sets of signal elements may correspond to different regions ofsome or all of an image, for example where the input data 202 comprisesimage or video data.

The input data 202 is processed to generate data 204 based on the inputdata 202. In this example, the data 204 is generated by downsampling theinput data 202 and is therefore referred to as “downsampled data”.

In this example, the downsampled data 204 is arranged as a 2×2 arraycomprising four signal elements. A downsampled signal element D_(i)corresponds to a downsampled version of the i^(th) set of signalelements in the input data 202.

The downsampled data 204 is a representation of the signal at arelatively low level of quality. The relatively low level of quality islow relative to one or more other levels of quality in the tieredhierarchy having multiple different levels of quality. The relativelylow level of quality may be the lowest level of quality in the tieredhierarchy or may be an intermediate level of quality.

The downsampled data 204 is processed to generate data 206. In thisexample, the data 206 is obtained by upsampling the downsampled data 204and is therefore referred to as “upsampled data”.

In this example, the upsampled data 206 is arranged as a 4×4 array ofsignal elements comprising four rows and four columns of signalelements. The upsampled data 206 comprises four sets of signal elements.In this example, each of the sets of signal elements is a 2×2 array ofsignal elements. A signal element U_(ij) corresponds to the j^(th)signal element in the i^(th) set of signal elements in the upsampleddata 206. The upsampled data 206 is based on a representation of thesignal at a relatively low level of quality, namely the downsampled data204.

The input data 202 and the upsampled data 206 are used to obtain a setof residual elements 208. The set of residual elements 208 is arrangedas a 4×4 array of signal elements comprising four rows and four columnsof signal elements. In this example, the set of residual elements 208comprises four groups of residual elements. Each of the groups ofresidual elements comprises a plurality of residual elements. In thisexample, each group of residual elements is a 2×2 array of residualelements. A residual element R_(ij) corresponds to the j^(th) residualelement in the i^(th) group of residual elements in the set of residualelements 208.

In this example, a given residual element in the set of residualelements 208 is obtained by comparing a value of a signal element in theupsampled data 206 with a value of a corresponding signal element in theinput data 202.

As such, the first apparatus 102 obtains a first plurality of residualelements, for example residual elements R₁₁, R₁₂, R₁₃, R₁₄. In thisexample, obtaining the first plurality of residual elements involves thefirst apparatus 102 generating or deriving the first plurality ofresidual elements, for example using the input data 202 and theupsampled data 206. In other examples, obtaining the first plurality ofresidual elements may involve the first apparatus 102 receiving thefirst plurality of residual elements or data useable to derive the firstplurality of residual elements.

The first apparatus 102 also obtains a second plurality of residualelements, for example residual elements R₂₁, R₂₂, R₂₃, R₂₄. In thisexample, obtaining the second plurality of residual elements involvesthe first apparatus 102 generating or deriving the second plurality ofresidual elements, for example using the input data 202 and theupsampled data 206. In other examples, obtaining the second plurality ofresidual elements may involve the first apparatus 102 receiving thesecond plurality of residual elements or data useable to derive thesecond plurality of residual elements.

The set of residual elements 208 is useable in combination with theupsampled data 206 to reconstruct the input data 202. For example, thecomparison between the input data 202 and the upsampled data 206 that isused to determine the set of residual elements 208 may be reversed suchthat the input data 202 can be obtained by comparing the upsampled data206 with the set of residual elements 208.

As such, the first plurality of residual elements is useable toreconstruct a first part of a first representation of a signal at therelatively high level of quality using a first part of a secondrepresentation of the signal at the relatively high level of quality.For example, residual elements R₁₁, R₁₂, R₁₃, R₁₄ are usable toreconstruct the part of the input data 202 consisting of signal elementsI₁₁, I₁₂, I₁₃, I₁₄ using the part of the upsampled data 206 consistingof signal elements U₁₁, U₁₂, U₁₃, U₁₄. The input data 202 and theupsampled data 206 are both at the relatively high level of quality. Thereconstruction may take place at the first apparatus 102, at the secondapparatus 104 or elsewhere. The first part of the second representationis derivable from a first set of one or more signal elements in arepresentation of the signal at the relatively low level of quality. Forexample, the part of the upsampled data 206 consisting of signalelements U₁₁, U₁₂, U₁₃, U₁₄ is derivable from signal element D₁ in thedownsamnpled data 204. The downsampled data 204 is a representation ofthe signal at the relatively low level of quality. The first set of oneor more signal elements in the representation of the signal at therelatively low level of quality would, in this example, consist solelyof signal element D₁.

The second plurality of residual elements is useable to reconstruct asecond part of the first representation of the signal at the relativelyhigh level of quality using a second, different part of the secondrepresentation of the signal at the relatively high level of quality.For example, residual elements R₂₁, R₂₂, R₂₃, R₂₄ are usable toreconstruct the part of the input data 202 consisting of signal elementsI₂₁, I₂₂, I₂₃, I₂₄ using the part of the upsampled data 206 consistingof signal elements U₂₁, U₂₂, U₂₃, U₂₄. The reconstruction may take placeat the first apparatus 102, at the second apparatus 104 or elsewhere.The second part of the second representation is derivable from a second,different set of one or more signal elements in the representation ofthe signal at the relatively low level of quality. For example, the partof the upsampled data 206 consisting of signal elements U₂₁, U₂₂, U₂₃,U₂₄ is derivable from signal element D₂ in the downsampled data 204. Thedownsampled data 204 is a representation of the signal at the relativelylow level of quality. The second set of one or more signal elements inthe representation of the signal at the relatively low level of qualitywould, in this example, consist solely of signal element D₂.

The first and second parts of the second representation of the signal atthe relatively high level of quality may be different in that at leastone of the first and second parts comprises at least one signal elementthat is not in the other of the first and second parts. The first andsecond parts of the second representation of the signal at therelatively high level of quality may be different in that the first andsecond parts do not comprise any signal elements in common.

The first and second sets of one or more signal elements in therepresentation of the signal at the relatively low level of quality maybe different in that at least one of the first and second sets comprisesat least one signal element that is not in the other of the first andsecond sets. The first and second sets of one or more signal elements inthe representation of the signal at the relatively low level of qualitymay be different in that the first and second sets do not comprise anysignal elements in common.

The set of residual elements 208 may be transmitted to the secondapparatus 104 to allow the second apparatus 104 to reconstruct the inputdata 202. The set of residual elements 208 may be encoded prior totransmission to reduce the amount of data involved in transmitting theset of residual elements 208 to the second apparatus 10.

In examples described herein, a set of correlation elements 210 isderived by performing at least one transformation operation involving atleast some residual elements in the set of residual elements 208. Thereader is referred to international patent application no.PCT/EP2013/059847, which published as WO2013/171173. PCT/EP2013/059847describes decomposition of residual data in a tiered hierarchy. Theentire contents of PCT/EP2013/059847 are incorporated herein byreference.

As such, the first apparatus 102 performs at least one transformationoperation. Multiple transformation operations may be performed, forexample in series and/or in parallel.

The at least one transformation operation involves at least one residualelement in the first plurality of residual elements, R₁₁, R₁₂, R₁₃, R₁₄,and at least one residual element in the second plurality of residualelements, R₂₁, R₂₂, R₂₃, R₂₄. The at least one transformation operationmay involve multiple residual elements in the first plurality ofresidual elements, R₁₁, R₁₂, R₁₃, R₁₄, and/or multiple residual elementsin the second plurality of residual elements. R₂₁, R₂₂, R₂₃, R₂₄. The atleast one residual element in the first plurality of residual elements,R₁₁, R₁₂, R₁₃, R₁₄, and the at least one residual element in the secondplurality of residual elements, R₂₁, R₂₂, R₂₃, R₂₄, may be inputs to theat least one transformation operation.

The at least one transformation operation may comprise at least onedirectional decomposition operation. A directional decompositionoperation may exploit correlation. For example, a directionaldecomposition operation may exploit directional (or ‘spatial’)correlation. This can lead to more efficient encoding, especially whenthere is a high degree of such correlation. A directional decompositionoperation may result in a set of directional components. Differentdirectional components may be encoded using different encodingparameters, for example according to the relative importance ofdifferent directional components in terms of perceived quality. Somedirectional components may therefore be encoded differently compared toother directional components without a noticeable difference in visualquality. This facilitates more selective encoding, for example to takeaccount of the importance of the individual directional components. Insome examples, the precision of encoding of some directional componentsis increased compared to the precision of encoding of other directionalcomponents, for example according to differences in perceptualimportance. In some examples, some directional components undergo morecompression compared to other directional components. In some examples,some directional components are allocated more bitrate compared to otherdirectional components, for example according to differences inperceptual importance between different directional components. Thereader is referred to international patent application no.PCT/EP2013/059847 for examples of how directional decompositioncomponents may be prioritised and encoded.

As is described in more detail below, multiple directional decompositionoperations may be performed to further exploit correlation betweendifferent sets and types of data elements. This may also lead to moreefficient encoding by providing more flexibility in terms of howindividual data elements are encoded by allowing groups of data elementsthat may otherwise all be encoded using the same encoder settings to bedecoupled from each other.

In this example, the set of correlation elements 210 is arranged as a4×4 array of correlation elements comprising four rows and four columnsof signal elements. In this example, the set of correlation elements 210comprises four groups of correlation elements. Each of the groups ofcorrelation elements comprises a plurality of correlation elements. Inthis example, each group of correlation elements is a 2×2 array ofcorrelation elements. A correlation element α_(ij) corresponds to the jcorrelation element in the i_(th) group of correlation elements in theset of correlation elements 210.

The set of correlation elements 210 may be derived by pre-multiplyingthe set of residual elements 208 with a first transformation (or‘transform’) matrix (or ‘kernel’), K₁. This may be written as α=K₁·R,where a represents the set of correlation elements 210 and R representsthe set of residual elements 208.

Instead of deriving the set of correlation elements 210 bypre-multiplying the entire set of residual elements 208 with a singlefirst transformation matrix, smaller groups of residual elements in theset of residual elements 208 could be pre-multiplied by respectivesmaller first transformation matrices. Using smaller matrices may resultin more efficient processing than using a single, larger matrix sincefewer calculations may be performed.

As such, the at least one transformation operation may comprise at leastone mathematical operation. The at least one transformation operationmay comprise at least one matrix multiplication operation. The at leastone transformation operation may comprise at least one linearcombination operation. For example, the at least one linear combinationoperation may be used instead of the at least one matrix multiplicationoperation. As such, the data to be encoded may be arranged in a formthat can lead to encoding in a more efficient and well-defined manner

In some examples, some or all of the correlation elements α_(ij) in theset of correlation elements 210 are derived based on residual elementsfrom only one of the pluralities of residual elements in the set ofresidual elements 208. Such correlation elements α_(ij) may beindicative of an extent of correlation between the residual elements onwhich they are based.

The set of correlation elements 210 may exploit correlation betweenresidual elements in the set of residual elements 208. Less data may beused to transmit the set of correlation elements 210 than the set ofresidual elements 208, particularly where there is strong correlation inthe set of residual elements 208. The set of correlation elements 210may exploit directional correlation. The set of correlation elements 210may exploit average, horizontal, vertical and/or diagonal correlation.In some examples, the set of correlation elements 210 includes at leastone value derived based on an average of at least some of the values inthe set of residual elements 208 and an average of at least some valuesin the downsampled data 204. Such a derived value may correspond to apredicted average value or a delta average value, Δ_(A), which isdescribed in international patent application no.

In this example, the set of correlation elements 210 is transformed intoa transformed set of correlation elements 212. In this example, thetransformed set of correlation elements 212 is derived by performing atleast one transformation operation involving some or all of thecorrelation elements in the set of correlation elements 210. Thetransformed set of correlation elements 212 may be derived bypre-multiplying the set of correlation elements 210 with a secondtransformation matrix (or ‘kernel’), K₂. This may be written as β=K₂·α,where β represents the transformed set of correlation elements 212 and αrepresents the set of correlation elements 210.

Instead of deriving the transformed set of correlation elements 212 bypre-multiplying the entire set of correlation elements 210 with a singlesecond transformation matrix, smaller groups of correlation elements inthe set of correlation elements 210 could be pre-multiplied byrespective smaller second transformation matrices.

In this example, the transformed set of correlation elements 212 isarranged as a 4×4 array of correlation elements comprising four rows andfour columns of signal elements. In this example, the transformed set ofcorrelation elements 212 comprises four groups of correlation elements.Each of the groups of correlation elements comprises a plurality ofcorrelation elements. In this example, each group of correlationelements is a 2×2 array of correlation elements. A correlation elementβ_(ij) corresponds to the j^(th) correlation element in the i^(th) groupof correlation elements in the set of correlation elements 212.

The transformed set of correlation elements 212 may facilitate moreefficient encoding compared to the set of correlation elements 210 byproviding a larger number of different types of correlation elementwhose associated encoding parameters may be set individually.

As such, the least one transformation operation may be performed byperforming a transformation (or ‘transform’) involving the firstplurality of residual elements, for example R₁₁, R₁₂, R₁₃, R₁₄, togenerate a first set of correlation elements, for example consisting ofα₁₁, β₁₂, α₁₃, α₁₄. Performing the least one transformation operationmay also involve performing a transformation involving the secondplurality of residual elements, for example R₂₁, R₂₂, R₂₃, R₂₄ togenerate a second set of correlation elements, for example consisting ofα₂₁, α₂₂, α₂₃, α₂₄. At least one correlation element in the first set ofcorrelation elements, for example α₁₁, α₁₂, α₁₃, α₁₄, is dependent on anextent of correlation between at least some of the first plurality ofresidual elements. R₁₁, R₁₂, R₁₃, R₁₄. Each of the correlation elementsin the first set of correlation elements, for example α₁₁, α₁₂, α₁₃,α₁₄, may be dependent on an extent of correlation between each of thefirst plurality of residual elements, R₁₁, R₁₂, R₁₃, R₁₄. At least onecorrelation element in the second set of correlation elements, forexample α₂₁, α₂₂, α₂₃, α₂₄ is dependent on an extent of correlationbetween at least some of the second plurality of residual elements, forexample R₂₁, R₂₂, R₂₃, R₂₄. Each of the correlation elements in thesecond set of correlation elements, for example α₂₁, α₂₂, α₂₃, α₂₄, maybe dependent on an extent of correlation between each of the secondplurality of residual elements, for example R₂₁, R₂₂, R₂₃, R₂₄. As such,in a first stage, residual elements are transformed into one or moreintermediate sets of correlation elements, for example the set ofcorrelation elements 210. Related correlation elements may be groupedtogether. For example, correlation elements may be grouped based on theresidual elements with which they are associated or based on correlationelement type.

The at least one transformation operation may be performed by performinga transformation involving at least one correlation element in a firstsubset of correlation elements in the set of correlation elements 210,for example α₁₁, α₁₂, α₁₃, α₁₄, and at least one correlation element ina second subset of correlation elements in the set of correlationelements 210, for example α₂₁, α₂₂, α₂₃, α₂₄, to generate at least onecorrelation element in a third set of correlation elements, for exampleβ₁₁ in the transformed set of correlation elements 212. For example,correlation elements α₁₁ and α₂₁ may be involved in a transformationoperation to generate correlation element β₁₁. One of more furthercorrelation elements, such as α₃₁ and α₄₁, may also be involved in atransformation operation to generate correlation element β₁₁. The atleast one correlation element, for example β₁₁, in the third set ofcorrelation elements, is dependent on an extent of correlation betweenthe at least one correlation element, for example α₁₁, in the first setof correlation elements and the at least one correlation element, forexample α₂₁, in the second set of correlation elements. As such, in asecond stage, correlation elements in one or more intermediate sets ofcorrelation elements, for example the set of correlation elements 210,are involved in a transformation operation into one or more final setsof correlation elements, for example the transformed set of correlationelements 212. The correlation elements in the one or more final sets ofcorrelation elements may exploit two levels of correlation, namely onebetween residual elements in the set of residual elements 208 andanother between correlation elements in the set of correlation elements210. This may lead to more efficient encoding where there is strongcorrelation at both levels and also allowing a fine-grainedconfiguration of encoder settings.

The at least one transformation operation may be performed by performinga different transformation involving the at least one correlationelement, for example α₁₁, in the first set of correlation elements andthe at least one correlation element, for example α₂₁, in the second setof correlation elements to generate at least one other correlationelement in the third set of correlation elements, for example β₂₁. Forexample, correlation elements α₁₁ and α₂₁ may be involved in atransformation operation using another type of transformation togenerate correlation element β₂₁. One or more further correlationelements, such as α₃₁ and α₄₁, may also be used to generate correlationelement β₂₁. The at least one other correlation element, for exampleβ₂₁, in the third set of correlation elements is dependent on an extentof correlation between the at least one correlation element, for exampleα₁₁, in the first set of correlation elements and the at least onecorrelation element, for example α₂₁, in the second set of correlationelements. Multiple different types of correlation may be exploited byperforming different transformations involving the intermediatecorrelation elements α_(ij). This may facilitate efficient encodingwhere multiple different types of correlation show strong correlationcharacteristics.

In this example, deriving the transformed set of correlation elements212 may comprise performing one or more transformation operationsinvolving at least one residual element, for example R₁₁, in the firstplurality of residual elements, R₁₁, R₁₂, R₁₃, R₁₄, and at least oneresidual element, for example R₂₁, in the second plurality of residualelements, R₂₁, R₂₂, R₂₃, R₂₄. A residual element, for example R₁₁, inthe first plurality of residual elements, R₁₁, R₁₂, R₁₃, R₁₄, may beused to generate a first correlation element, for example α₁₁, in theset of correlation elements 210 and a residual element, for example R₂₁,in the second plurality of residual elements may be used to generate asecond correlation element, for example α₂₁, in the set of correlationelements 210. The first and second correlation elements, for example α₁₁and α₂₁, in the set of correlation elements 210 may be used to generateat least one correlation element, for example β₁₁, in the transformedset of correlation elements 212.

In contrast to the correlation elements in the transformed set ofcorrelation elements 212, in this example the correlation elements inthe set of correlation elements 210 are not generated by performing oneor more transformation operations involving at least one residual in thefirst plurality of residual elements, R₁₁, R₁₂, R₁₃, R₁₄, and at leastone residual element in the second plurality of residual elements. R₂₁,R₂₂, R₂₃, R₂₄. Instead, in this example, each of the correlationelements in the set of correlation elements 210 is generated byperforming one or more transformation operations involving at least oneresidual from a single plurality of residual elements only, for exampleinvolving only residual elements in the first plurality of residualelements, R₁₁, R₁₂, R₁₃, R₁₄.

In this example, the at least one transformation operation generates atleast one correlation element, for example correlation element β₁₁ inthe transformed set of correlation elements 212. The at least onecorrelation element is dependent on an extent of correlation between theat least one residual element in the first plurality of residualelements and the at least one residual element in the second pluralityof residual elements. The value of the at least one correlation elementmay be inversely proportional to the extent of correlation between theat least one residual element in the first plurality of residualelements and the at least one residual element in the second pluralityof residual elements. In other words, the stronger the correlationbetween the at least one residual element in the first plurality ofresidual elements and the at least one residual element in the secondplurality of residual elements the closer the value of the at least onecorrelation element is to zero. The extent of correlation may be relatedto the amount or strength of correlation.

In some examples, each of the correlation elements in the transformedset of correlation elements 212 exploits correlation between each of theresidual elements in the set of residual elements 208. As such the atleast one transformation operation may involve all of the residualelements in the first plurality of residual elements, R₁₁, R₁₂, R₁₃,R₁₄, and all of the residual elements in the second plurality ofresidual elements. R₂₁, R₂₂, R₂₃, R₂₄. Correlation between a relativelylarge number of residual elements may therefore be exploited. This maylead to efficient encoding where there is a strong correlation between arelatively large number of correlation elements.

A plurality of correlation elements may be generated in performing theat least one transformation operation. By generating multiplecorrelation elements, different types of correlation may be exploited.As such, more efficient encoding can be obtained where correlation isstrong in different directions. Furthermore, different encoder settingsmay be used for different correlation elements. This facilitates moreselective encoding, for example to take account of the importance of theindividual correlation elements.

In some examples, the transformed set of correlation elements 212 isused to obtain one or more further transformed sets of correlationelements (not shown). The one or more further transformed sets ofcorrelation elements may be obtained by performing one or more transformoperations on the transformed set of correlation elements 212, forexample in series. The one or more transform operations may involve oneor more directional decomposition operations, as described above. Assuch, the one or more further transformed sets of correlation elementsmay be indicative of an extent of correlation, for example a directionalor spatial correlation, between different correlation elements in thetransformed set of correlation elements 212, or in sets of data elementsderived therefrom. This can lead to more efficient encoding, especiallywhen there is a high degree of such correlation.

The first apparatus 102 performs the at least one transformationoperation prior to the at least one correlation element being encoded.The first apparatus 102 may encode the at least one correlation element.The at least one correlation element may be encoded by one or moreentities other than the first apparatus 102.

The at least one correlation element being encoded may comprisequantisation being performed on the at least one correlation element.Quantisation facilitates reducing the amount of data used to representthe at least one correlation element. The first apparatus 102 mayquantise the at least one correlation element. The at least onecorrelation element may be quantised by one or more entities other thanthe first apparatus 102.

Individual encoding parameters may be used for encoding individualcorrelation elements in the transformed set of correlation elements 212.For example, correlation elements that are relatively unimportant(relative to other correlation elements in the transformed set ofcorrelation elements 212) may be subject to a higher level of encoding,for example quantisation, than correlation elements that are relativelyimportant (relative to other correlation elements in the transformed setof correlation elements 212).

In this example, the encoder device 108 obtains a set of encodingparameters. Each of the correlation elements in the plurality ofcorrelation elements may be associated with one or more encodingparameters. This allows for fine-grained specification of encodingparameters, for example to achieve an overall bit rate allowance for thetransformed set of correlation elements 212. The individual encodingparameters may also be indicative of the importance of the associatedcorrelation element. This may facilitate efficient encoding byprioritising the most important correlation elements. A correlationelement may be associated with multiple encoding parameters. An encodingparameter may be associated with multiple correlation elements.

In some examples, an encoding parameter or other data indicates that anassociated correlation element is not to be encoded. For example, theassociated correlation element may be discarded and not encoded at all.In some examples, there is a predefined importance threshold level belowwhich a correlation element is not encoded. Instead, a default value forthat correlation element may be used. Data may therefore be generated toindicate that at least some of the plurality of correlation elements arenot to be encoded. Encoding may be made more efficient by not encodingsome of the plurality of correlation elements. Encoding may be moreefficient as less data is encoded.

The encoding parameter for each of the correlation elements in theplurality of correlation elements may be determined. Determining theencoding parameters may involve looking up some or all of the encodingparameters, calculating the some or all of the encoding parameters orobtaining some or all of the encoding parameters in another way.

In some examples, the value of an encoding parameter is determined basedat least in part on the importance of the correlation element with whichit is associated, relative to the importance other correlation elementsin the transformed set of correlation elements 212. The value of theencoding parameter may be indicative of the importance of the associatedcorrelation element in terms of perceived quality.

In some examples, at least some of the correlation elements in thetransformed set of correlation elements 212 are associated with the sameencoding parameter and/or encoding parameter value as each other. Thismay indicate that those correlation elements are all equally importantin terms of perceived quality. Encoding may be enhanced by designatingthe same level of encoding for different ones of the correlationelements, for example where any differences in importance of associatedcorrelation elements to not exceed a threshold level, so the encoder canuse the same encoder settings for multiple ones of the correlationelements. The threshold level may be zero or non-zero.

In some examples, at least some of the correlation elements in thetransformed set of correlation elements 212 are associated withdifferent encoding parameters and/or encoding parameter values. This mayfacilitate encoding by determining the relative importance of thosecorrelation elements in terms of perceived quality. For example,different amounts of quantisation may be used dependent upon therelative importance of those correlation elements in terms of perceivedquality.

In some examples, all of the correlation elements in the transformed setof correlation elements are associated with different encodingparameters and/or encoding parameter values.

Various different factors may be taken into account in determining thevalues of the encoding parameters in the set of encoding parameters.Examples of such factors include, but are not limited to, frequencyanalysis, angular frequency analysis, viewing distance analysis andpixel size analysis. Taking such factors into account may facilitateranking of the correlation elements in terms of perceptual importance.This, in turn, may be used to determine the extent to which, forexample, quantisation should be applied to the correlation elements.

At least some of the encoding parameters may be representative ofrespective measures of perceptual importance amongst the correlationelements. This may enhance encoding by allowing encoding to be weightedbased on a measure of importance of the correlation elements.

At least some of the encoding parameters may be the same for differentones of the correlation elements in the plurality of correlationelements.

The plurality of encoding parameters may comprise at least onequantisation coefficient. As such, different levels of quantisation maybe applied to different correlation elements, for example based on theirimportance. This may provide encoding techniques in which reducing anamount of data to be transmitted is balanced against the perception ofquality associated with quantised data.

In some examples, the first apparatus 102 outputs the set of encodingparameters to at least one further apparatus, for example the secondapparatus 104. The second apparatus 104 may use the set of encodingparameters to facilitate decoding of the encoded correlation elements.As such, data comprising the encoding parameters may be output. Thisalso facilitates encoding where encoding is performed by another entity.

The plurality of correlation elements may be encoded based on the set ofencoding parameters. As such, a reliable mechanism is provided forencoding the plurality of correlation elements, for example to reflecttheir relative importance.

In this example, the first apparatus 102 performs the at least onetransformation operation, to generate the at least one correlationelement in the transformed set of correlation elements, prior to the atleast one correlation element being encoded.

In this example, the first apparatus 102 transmits the downsampled data204 and the transformed set of correlation elements 212 to the secondapparatus 104, possibly in an encoded form.

Referring to FIG. 3, there is shown schematically an example of a signalprocessing technique 300. The signal processing technique 300 may beperformed by the second apparatus 104. Some items depicted in FIG. 3 aresimilar to items shown in FIG. 2. Corresponding reference signs,incremented by 100, have therefore been used for similar items.

In this example, the second apparatus 104 receives downsampled data 304and the transformed set of correlation elements 312 from the firstapparatus 102. In other examples, the second apparatus 104 receives datausable to obtain the downsampled data 304 and/or the transformed set ofcorrelation elements 312, rather than the downsampled data 304 and/orthe transformed set of correlation elements 312 itself. For example, thesecond apparatus 104 may derive the downsampled data 304 and/or thetransformed set of correlation elements 312 based on the received data.

Pre-multiplying both sides of the above equation β=K₂·α by the inverseof K₂, namely K₂ ⁻¹, gives K₂ ⁻¹·β=K₂ ⁻¹·K₂·α. Since pre-multiplying amatrix by its inverse gives the identity matrix, K₂ ⁻¹·β=α. As such, thesecond apparatus 104 can derive the set of correlation elements 310 bypre-multiplying the transformed set of correlation elements 312 with theinverse of the second transformation matrix, K₂ ⁻¹. Pre-multiplying bothsides of the above equation α=K₁·R by the inverse of K₁, namely K₁ ⁻¹,gives K₁·α=K₁ ⁻¹·K₁·R. Since pre-multiplying a matrix by its inversegives the identity matrix, K₁ ⁻¹·α=R. As such, the second apparatus 104can derive the set of residual elements 308 by pre-multiplying the setof correlation elements 310 with the inverse of the first transformationmatrix, K₁ ⁻¹.

As such, the second apparatus 104 recovers the set of residual elements308.

The downsampled data 304 is processed to generate data 306. In thisexample, the data 306 is obtained by upsampling the downsampled data 304in the same manner as the first apparatus 102 upsampled the data 204. Inthis specific example, the data 306 is referred to as “upsampled data”,it being understood that the data could be processed in a different wayin other examples.

The second apparatus 104 can therefore recover the input data 302 usingthe upsampled data 306 and the set of residual elements 308 since all ofthe values U_(ij) and R_(ij) are known to the second apparatus 104.

The second apparatus 104 performs at least one transformation operationinvolving at least one correlation element, for example β₁₁ in thetransformed set of correlation elements 312. The at least onetransformation operation may involve at least one transformation that isthe inverse of a transformation performed by the first apparatus 102.The at least one transformation operation generates at least oneresidual element, for example R₁₁, in a first plurality of residualelements, for example R₁₁, R₁₂, R₁₃, R₁₄, and at least one residualelement, for example R₂₁, in a second plurality of residual elements,for example R₂₁, R₂₂, R₂₃, R₂₄. The at least one transformationoperation may generate one or more further residual elements. The atleast one transformation operation may involve one or more additionalcorrelation elements.

The at least one correlation element, for example β₁₁, is dependent onan extent of correlation between the at least one residual element, forexample R₁₁, in the first plurality of residual elements and the atleast one residual element, for example R₂₁, in the second plurality ofresidual elements. The first plurality of residual elements, for exampleR₁₁, R₁₂, R₁₃, R₁₄, is useable to reconstruct a first part of a firstrepresentation of a signal at a relatively high level of quality, forexample the part of the input data 302 consisting of signal elementsI₁₁, I₁₂, I₁₃, I₁₄, using a first part of a second representation of thesignal at the relatively high level of quality, for example the part ofthe upsampled data 306 consisting of signal elements U₁₁, U₁₂, U₁₃, U₁₄.The second plurality of residual elements, for example R₂₁, R₂₂, R₂₃,R₂₄, is useable to reconstruct a second, different part of the firstrepresentation of the signal, for example the part of the input data 302consisting of signal elements I₂₁, I₂₂, I₂₃, I₂₄, using a second,different part of the second representation of the signal, for examplethe part of the upsamnpled data 306 consisting of signal elements U₂₁,U₂₂, U₂₃, U₂₄.

The second apparatus 104 derives the first part of the secondrepresentation, for example the part of the upsampled data 306consisting of signal elements U₁₁, U₁₂, U₁₃, U₁₄, from a first set ofone or more signal elements in a representation of the signal at arelatively low level of quality, for example signal element D₁ in thedownsampled data 304.

The second apparatus 104 derives the second, different part of thesecond representation, for example the part of the upsampled data 306consisting of signal elements U₂₁, U₂₂, U₂₃, U₂₄, from a second,different set of one or more signal elements in a representation of thesignal at a relatively low level of quality, for example signal elementD₂ in the downsampled data 304.

The second apparatus 104 uses at least the first plurality of residualelements, for example R₁₁, R₁₂, R₁₃, R₁₄, the second plurality ofresidual elements, for example R₂₁, R₂₂, R₂₃, R₂₄, the first part of thesecond representation, for example U₁₁, U₁₂, U₁₃, U₁₄, and the secondpart of the second representation, for example U₂₁, U₂₂, U₂₃, U₂₄, togenerate output data. The output data may comprise input data 302.

The second apparatus 104 is configured to perform the at least onetransformation operation following the at least one correlation element,for example β₁₁, having been decoded. The at least one correlationelement may be decoded by the second apparatus 104 or by one or moreother entities.

The second apparatus 104 may have at least some other features describedherein, for example features described in relation to the firstapparatus 102. The second apparatus 104 may be configured to perform atleast some other techniques described herein, for example techniquesdescribed herein in relation to the first apparatus 102.

Referring to FIG. 4, there is shown a schematic block diagram of anexample of an apparatus 400.

In an example, the apparatus 400 comprises a decoder device. In anotherexample, the apparatus 400 comprises an encoder device.

Other examples of apparatus 400 include, but are not limited to, amobile computer, a personal computer system, a wireless device, basestation, phone device, desktop computer, laptop, notebook, netbookcomputer, mainframe computer system, handheld computer, workstation,network computer, application server, storage device, a consumerelectronics device such as a camera, camcorder, mobile device, videogame console, handheld video game device, a peripheral device such as aswitch, modem, router, etc., or in general any type of computing orelectronic device.

In this example, the apparatus 400 comprises one or more processors 401configured to process information and/or instructions. The one or moreprocessors 401 may comprise a central processing unit (CPU). The one ormore processors 401 are coupled with a bus 402. Operations performed bythe one or more processors 401 may be carried out by hardware and/orsoftware. The one or more processors 401 may comprise multipleco-located processors or multiple disparately located processors.

In this example, the apparatus 400 comprises computer-useable volatilememory 403 configured to store information and/or instructions for theone or more processors 401. The computer-useable volatile memory 403 iscoupled with the bus 402. The computer-useable volatile memory 403 maycomprise random access memory (RAM).

In this example, the apparatus 400 comprises computer-useablenon-volatile memory 404 configured to store information and/orinstructions for the one or more processors 401. The computer-useablenon-volatile memory 404 is coupled with the bus 402. Thecomputer-useable non-volatile memory 404 may comprise read-only memory(ROM).

In this example, the apparatus 400 comprises one or more data-storageunits 405 configured to store information and/or instructions. The oneor more data-storage units 405 are coupled with the bus 402. The one ormore data-storage units 405 may for example comprise a magnetic oroptical disk and disk drive or a solid-state drive (SSD).

In this example, the apparatus 400 comprises one or more input/output(I/O) devices 406 configured to communicate information to and/or fromthe one or more processors 401. The one or more I/O devices 406 arecoupled with the bus 402. The one or more I/O devices 406 may compriseat least one network interface. The at least one network interface mayenable the apparatus 400 to communicate via one or more datacommunications networks. Examples of data communications networksinclude, but are not limited to, the Internet and a Local Area Network(LAN). The one or more I/O devices 406 may enable a user to provideinput to the apparatus 400 via one or more input devices (not shown).The one or more input devices may include for example a remote control,one or more physical buttons etc. The one or more I/O devices 406 mayenable information to be provided to a user via one or more outputdevices (not shown). The one or more output devices may for exampleinclude a display screen.

Various other entities are depicted for the apparatus 400. For example,when present, an operating system 407, signal processing module 408, oneor more further modules 409, and data 410 are shown as residing in one,or a combination, of the computer-usable volatile memory 403,computer-usable non-volatile memory 404 and the one or more data-storageunits 405. The signal processing module 408 may be implemented by way ofcomputer program code stored in memory locations within thecomputer-usable non-volatile memory 404, computer-readable storage mediawithin the one or more data-storage units 405 and/or other tangiblecomputer-readable storage media. Examples of tangible computer-readablestorage media include, but are not limited to, an optical medium (e.g.,CD-ROM. DVD-ROM or Blu-ray), flash memory card, floppy or hard disk orany other medium capable of storing computer-readable instructions suchas firmware or microcode in at least one ROM or RAM or Programmable ROM(PROM) chips or as an Application Specific Integrated Circuit (ASIC).

The apparatus 400 may therefore comprise a signal processing module 408which can be executed by the one or more processors 401. The signalprocessing module 408 can be configured to include instructions toimplement at least some of the operations described herein. Duringoperation, the one or more processors 401 launch, run, execute,interpret or otherwise perform the instructions in the signal processingmodule 408.

Although at least some aspects of the examples described herein withreference to the drawings comprise computer processes performed inprocessing systems or processors, examples described herein also extendto computer programs, for example computer programs on or in a carrier,adapted for putting the examples into practice. The carrier may be anyentity or device capable of carrying the program.

It will be appreciated that the apparatus 400 may comprise more, fewerand/or different components from those depicted in FIG. 4.

The apparatus 400 may be located in a single location or may bedistributed in multiple locations. Such locations may be local orremote.

The techniques described herein may be implemented in software orhardware, or may be implemented using a combination of software andhardware. They may include configuring an apparatus to carry out and/orsupport any or all of techniques described herein.

Various measures (for example apparatuses, methods, computer programsand computer-readable media) are provided in which a first plurality ofresidual elements is obtained. The first plurality of residual elementsis useable to reconstruct a first part of a first representation of asignal at a relatively high level of quality using a first part of asecond representation of the signal at the relatively high level ofquality. The first part of the second representation is derivable from afirst set of one or more signal elements in a representation of thesignal at a relatively low level of quality. A second plurality ofresidual elements is obtained. The second plurality of residuals isuseable to reconstruct a second, different part of the firstrepresentation of the signal using a second, different part of thesecond representation of the signal. The second part of the secondrepresentation is derivable from a second, different set of one or moresignal elements in the representation of the signal at the relativelylow level of quality. At least one transformation operation involving atleast one residual element in the first plurality of residual elementsand at least one residual element in the second plurality of residualelements is performed to generate at least one correlation element. Theat least one correlation element is dependent on an extent ofcorrelation between the at least one residual element in the firstplurality of residual elements and the at least one residual element inthe second plurality of residual elements. The apparatus is configuredto perform the at least one transformation operation prior to the atleast one correlation element being encoded.

In examples described above, the at least one transformation operationcomprises at least one directional decomposition operation.

In examples described above, the at least one transformation operationcomprises at least one mathematical operation.

In examples described above, the at least one transformation operationcomprises at least one linear combination operation.

In examples described above, the at least one transformation operationcomprises at least one linear combination operation.

In examples described above, a plurality of correlation elements isgenerated in performing the at least one transformation operation.

In examples described above, data is generated to indicate that at leastsome of the plurality of correlation elements are not to be encoded.

In examples described above, each of the correlation elements in theplurality of correlation elements is associated with an encodingparameter.

In examples described above, the encoding parameter for each of thecorrelation elements in the plurality of correlation elements isdetermined.

In examples described above, the encoding parameter for each of thecorrelation elements in the plurality of correlation elements isdetermined based on a position of the correlation elements in theplurality of correlation elements.

In examples described above, a value of an encoding parameter in theplurality of correlation elements is independent of a value of acorrelation element associated with the encoding parameter.

In examples described above, at least some of the encoding parametersare different for different ones of the correlation elements in theplurality of correlation elements.

In examples described above, at least some of the encoding parametersare the same for different ones of the correlation elements in theplurality of correlation elements.

In examples described above, at least some of the encoding parametersare representative of respective measures of perceptual importanceamongst the correlation elements.

In examples described above, the plurality of encoding parameterscomprises at least one quantisation coefficient.

In examples described above, the plurality of correlation elements isencoded based on the encoding parameters.

In examples described above, data comprising the encoding parameters isoutput.

In examples described above, the at least one correlation element beingencoded comprises quantisation being performed on the at least onecorrelation element.

In examples described above, at least one transformation operation isperformed by performing a transformation involving the first pluralityof residual elements to generate a first set of correlation elements andperforming a transformation involving the second plurality of residualelements to generate a second set of correlation elements. At least onecorrelation element in the first set of correlation elements isdependent on an extent of correlation between at least some of the firstplurality of residual elements. At least one correlation element in thesecond set of correlation elements is dependent on an extent ofcorrelation between at least some of the second plurality of residualelements.

In examples described above, different correlation elements in the firstand second sets of correlation elements represent different types ofcorrelation. Correlation elements of the same type are grouped togetherwithin the first and second sets of correlation elements.

In examples described above, the at least one transformation operationis performed by performing a transformation involving at least onecorrelation element in the first set of correlation elements and atleast one correlation element in the second set of correlation elementsto generate at least one correlation element in a third set ofcorrelation elements. The at least one correlation element in the thirdset of correlation elements is dependent on an extent of correlationbetween the at least one correlation element in the first set ofcorrelation elements and the at least one correlation element in thesecond set of correlation elements.

In examples described above, the at least one transformation operationis performed by performing a different transformation involving the atleast one correlation element in the first set of correlation elementsand the at least one correlation element in the second set ofcorrelation elements to generate at least one other correlation elementin the third set of correlation elements. The at least one othercorrelation element in the third set of correlation elements isdependent on an extent of correlation between the at least onecorrelation element in the first set of correlation elements and the atleast one correlation element in the second set of correlation elements.

In examples described above, at least one transformation operationinvolves all of the residual elements in the first plurality of residualelements and all of the residual elements in the second plurality ofresidual elements.

Various measures (for example apparatuses, methods, computer programsand computer-readable media) are provided in which at least onetransformation operation involving at least one correlation element isperformed to generate at least one residual element in a first pluralityof residual elements and at least one residual element in a secondplurality of residual elements. The at least one correlation element isdependent on an extent of correlation between the at least one residualelement in the first plurality of residual elements and the at least oneresidual element in the second plurality of residual elements. The firstplurality of residual elements is useable to reconstruct a first part ofa first representation of a signal at a relatively high level of qualityusing a first part of a second representation of the signal at therelatively high level of quality. The second plurality of residualelements is useable to reconstruct a second, different part of the firstrepresentation of the signal using a second, different part of thesecond representation of the signal. The first part of the secondrepresentation is derived from a first set of one or more signalelements in a representation of the signal at a relatively low level ofquality. The second part of the second representation is derived from asecond, different set of one or more signal elements in therepresentation of the signal at the relatively low level of quality. Atleast the first plurality of residual elements, the second plurality ofresidual elements, the first part of the second representation and thesecond part of the second representation are used to generate outputdata. The at least one transformation operation is performed followingthe at least one correlation element being decoded.

Various measures (for example apparatuses, methods, computer programsand computer-readable media) are provided to process signal data. A setof residual elements is obtained. A first plurality of residual elementsin the set of residual elements is useable to generate a first set ofcorrelation elements. A second plurality of residual elements in the setof residual elements is useable to generate a second set of correlationelements. At least one correlation element in the first set ofcorrelation elements is indicative of an extent of correlation, forexample spatial or directional correlation, between at least some of thefirst plurality of residual elements. At least one correlation elementin the second set of correlation elements is indicative of an extent ofcorrelation, for example spatial or directional correlation, between atleast some of the second plurality of residual elements. The first andsecond sets of correlation elements are transformable into a third setof correlation elements. At least one correlation element in the thirdset of correlation elements is indicative of an extent of correlation,for example spatial or directional correlation, between one or more ofthe at least one correlation elements in the first set of correlationelements and one or more of the at least one correlation elements in thesecond set of correlation elements. A first correlation element in thethird set of correlation elements is generated by performing a firsttransformation operation involving at least one of the first pluralityof residual elements and at least one of the second plurality ofresidual elements. A second correlation element in the third set ofcorrelation elements is generated by performing a second transformationoperation on the at least one of the first plurality of residualelements and on the at least one of the second plurality of residualelements.

The above embodiments are to be understood as illustrative examples.Further embodiments are envisaged.

In examples described above, the second apparatus 104 is able to recoveror reconstruct fully the input data obtained by the first apparatus 102.In other examples, the second apparatus 104 may not be able to recoverfully the input data obtained by the first apparatus 102. For example,some information that may be needed to recover the original input datamay be lost during quantisation and may not be recoverable by the secondapparatus 104.

In examples described above, the transformed set of correlation elements212 is obtained indirectly from the set of residual elements 208 by twoseparate matrix multiplications involving the first and secondtransformation matrices K₁ and K₂. In other examples, the transformedset of correlation elements 212 is obtained directly from the set ofresidual elements 208 by combining the two matrix multiplicationsinvolving the first and second transformation matrices K₁ and K₂described above. In such other examples, rather than carrying out twoseparate matrix multiplications, one involving K₁ and the otherinvolving K₂, a single matrix multiplication operation merging bothmatrix transformations is used. Pre-multiplying both sides of the aboveequation α=K₁·R by K₂ gives K₂·α=K₂·K₁·R. Using the above equation,β=K₂·α, it can be seen that β=K₂·β=K₂·K₁·R. A merged matrix, K₃, may beobtained by K₃=K₂·K₁ and the transform performed as β=K₃·R. Merging twomatrix multiplications may be beneficial in that memory is not used tostore an intermediate matrix. Processing time may also be reduced inperforming one matrix multiplication compared to performing two matrixmultiplications.

In examples described above, a set of correlation elements is obtainedby pre-multiplying a set of residual elements with a firsttransformation matrix, K₁, and a transformed set of correlation elementsis obtained by pre-multiplying the set of correlation elements with asecond transformation matrix, K₂. The set of correlation elements andthe transformed set of correlation elements are derived from residualelements corresponding to a 4×4 array of signal elements. As such, eachof K₁ and K₂ may comprise a 16×16 transformation matrix which is appliedto a 16×1 set of data elements. In some examples, smaller groups ofresidual elements and/or correlation elements are pre-multiplied byrespective smaller transformation matrices. For example, one or both ofK₁ and K₂ may comprise a 4×4 transformation matrix and may be applied toa 4×1 set of data elements.

In one example, the first transformation matrix, K₁, comprises thefollowing matrix:

$K_{1} = {{\frac{1}{4}\begin{bmatrix}2 & 2 & 0 & 0 \\1 & {- 1} & 1 & {- 1} \\0 & 0 & 2 & 2 \\1 & {- 1} & {- 1} & 1\end{bmatrix}}.}$

Applying K, to four 4×1 sets of residual elements, each setcorresponding to a 2×2 array of signal elements, results in four 4×1sets of correlation elements. At least one of the correlation elementsin each set is derived from at least two residual elements associatedwith signal elements from one row of signal elements and a differentnumber of residual elements associated with signal elements from anotherrow of signal elements. For example, a correlation element resultingfrom the first row of K, is derived from two residual elementsassociated with signal elements from one row of signal elements and noresidual elements associated with signal elements from another row ofsignal elements. A correlation element resulting from the second row ofK₁, on the other hand, is derived from two residual elements associatedwith signal elements from one row of signal elements and two residualelements associated with signal elements from another row of signalelements. K, may be referred to as a “1D” correlation transformationmatrix in this example, in that it emphasises horizontal correlationover vertical correlation.

As such, in this example, at least one correlation element in each setof correlation elements is not derived from contributions of equalnumbers of residual elements associated with signal elements from thefirst and second rows of signal elements of the input data. By takinguneven contributions in this way, the contributions of residual elementsassociated with signal elements from different rows of signal elementsof the input data in deriving the correlation elements can be morereadily weighted. In effect, the rows of signal elements of the inputdata may be decoupled in terms of their relative contributions to theset of correlation elements. The elements in the transformation matrix.K₁, may be selected to influence the relative contributions of residualelements associated with signal elements from different rows of signalelements in deriving the set of correlation elements.

Decoupling the rows of signal elements of the input data in terms oftheir respective contributions to the set of correlation elementsprovides various effects. Allowing different rows of input signalelements to have different weighted contributions may, for example,provide an improved performance when handling interlaced video signals.In interlaced video, a given video frame, for example comprising1920×1080 data elements, is an arrangement of two fields of video data,each comprising 1920×540 data elements, whose rows of data elements areinterlaced with each other. Each of the two fields is representative ofthe video at different time samples. Accordingly, vertical correlationbetween adjacent signal elements in a frame of interlaced video signal,that is, correlation between data elements of adjacent rows, is reduced,and may even be absent or artificial. Horizontal correlation betweenadjacent data elements, that is, correlation between data elements ofadjacent columns, is more significant than vertical correlation.Therefore, the ability to influence the contributions of data elementsfrom different rows so that, for example, only data elements from asingle field are used, may be beneficial in ensuring that any false orunwanted vertical correlation is not taken into account. Additionally,downsampling and upsampling interlaced video signals in two-dimensionsmay introduce further considerations, owing to the two interlaced fieldscorresponding to two different points in time. The fields may bede-interlaced prior to downsampling, but this may be computationallyexpensive and may produce unwanted image artefacts or discontinuities.Therefore, the interlaced video may be downsampled and upsampled in thehorizontal dimension only. An example resolution of a field ofinterlaced video is 1920×540. A horizontally downsampled rendition ofthe field of interlaced video may have a resolution of 960×540, forexample. In such a horizontal downsampling operation, or a correspondinghorizontal upsampling operation, no contributions from verticallyadjacent data elements are taken into account.

Furthermore, taking unequal contributions from different rows of signalelements may also be beneficial in the case of progressive video. Inprogressive video, cameras may scan images horizontally and maytherefore have an effectively one-dimensional transfer function.Therefore, any vertical correlation may be false and it may be desirableto reduce the influence of such artificial vertical correlation.

Each of the four 4×1 sets of correlation elements may be pre-multipliedby a second 4×4 transformation matrix, K₂, to obtain four sets of fourtransformed correlation elements. In an example, the secondtransformation matrix, K₂, comprises the following matrix:

$K_{2} = {{\frac{1}{4}\begin{bmatrix}1 & 1 & 1 & 1 \\1 & {- 1} & 1 & {- 1} \\1 & 1 & {- 1} & {- 1} \\1 & {- 1} & {- 1} & 1\end{bmatrix}}.}$

The second transformation matrix, K₂, may be referred to as a “2D”correlation transformation matrix in this example, in that it treatshorizontal and vertical correlation equally. For example, eachtransformed correlation element in a set of transformed correlationelements may take equal contributions from every correlation element ina corresponding set of correlation elements.

The first transformation matrix, K₁, and/or the second transformationmatrix, K₂, may be different from those shown above. In some examples,the first transformation matrix, K₁, is the same as the secondtransformation matrix, K₂. In some examples, the first transformationmatrix. K₁, is a “2D” correlation transformation matrix as describedabove. In some examples, the second transformation matrix, K₂, is a “1D”correlation transformation matrix as described above.

In some examples, instead of K₁ and K₂ each being applied to four setsof four elements, a set of sixteen correlation elements is derived byapplying a first 16×16 transformation matrix to a set of residualelements corresponding to a 4×4 array of signal elements, and a set ofsixteen transformed correlation elements is then derived by applying asecond 16×16 transformation matrix to the set of sixteen correlationelements. The first 16×16 transformation matrix may relate to a 1Dcorrelation transformation and the second 16×16 transformation matrixmay relate to a 2D correlation transformation.

It is to be understood that any feature described in relation to any oneembodiment may be used alone, or in combination with other featuresdescribed, and may also be used in combination with one or more featuresof any other of the embodiments, or any combination of any other of theembodiments. Furthermore, equivalents and modifications not describedabove may also be employed without departing from the scope of theinvention, which is defined in the accompanying claims.

1. An apparatus comprising: at least one processor, and non-transitorysystem memory having stored thereon computer-executable instructionswhich, when executed by the at least one processor, cause the apparatusto perform the following: obtain a first plurality of residual elementsuseable to reconstruct a first part of a first representation of asignal at a relatively high level of quality using a first part of asecond representation of the signal at the relatively high level ofquality, the first part of the second representation being derivablefrom a first set of one or more signal elements in a representation ofthe signal at a relatively low level of quality; obtain a secondplurality of residual elements useable to reconstruct a second,different part of the first representation of the signal using a second,different part of the second representation of the signal, the secondpart of the second representation being derivable from a second,different set of one or more signal elements in the representation ofthe signal at the relatively low level of quality; and perform at leastone transformation operation involving at least one residual element inthe first plurality of residual elements and at least one residualelement in the second plurality of residual elements to generate aplurality of correlation elements, the plurality of correlation elementsbeing dependent on an extent of correlation between the first pluralityof residual elements and the second plurality of residual elements,wherein the at least one transformation operation is performed prior tothe plurality of correlation elements being encoded, wherein the atleast one transformation operation implements at least one matrixmultiplication, and wherein the at least one matrix multiplicationimplements a first transformation operation applied to the firstplurality of residual elements and the second plurality of residualelements and a second transformation operation applied to a result ofthe first transformation.
 2. The apparatus according to claim 1, whereinthe at least one transformation operation comprises at least onedirectional decomposition operation.
 3. The apparatus according to claim2, wherein the first transformation operation and the secondtransformation both comprise directional decompositions, the directionaldecompositions generating decomposed elements representing each ofaverage, horizontal, vertical and diagonal directions.
 4. The apparatusaccording to claim 1, wherein the computer-executable instructions causethe apparatus to generate data to indicate that at least some of theplurality of correlation elements are not to be encoded.
 5. Theapparatus according to claim 1, wherein each of the correlation elementsin the plurality of correlation elements is associated with an encodingparameter and the computer-executable instructions cause the apparatusto determine the encoding parameter for each of the correlation elementsin the plurality of correlation elements.
 6. The apparatus according toclaim 5, wherein the computer-executable instructions cause theapparatus to determine the encoding parameter for each of thecorrelation elements in the plurality of correlation elements based on aposition of the correlation elements in the plurality of correlationelements.
 7. The apparatus according to claim 6, wherein a value of anencoding parameter in the plurality of correlation elements isindependent of a value of a correlation element associated with theencoding parameter.
 8. The apparatus according to claim 5, wherein atleast some of the encoding parameters are representative of respectivemeasures of perceptual importance amongst the correlation elements. 9.The apparatus according to claim 1, wherein the computer-executableinstructions cause the apparatus to perform the at least onetransformation operation by: performing the first transformation on thefirst plurality of residual elements to generate a first set ofcorrelation elements, at least one correlation element in the first setof correlation elements being dependent on an extent of correlationbetween at least some of the first plurality of residual elements;performing the first transformation on the second plurality of residualelements to generate a second set of correlation elements, at least onecorrelation element in the second set of correlation elements beingdependent on an extent of correlation between at least some of thesecond plurality of residual elements; and performing the secondtransformation on the first and second set of correlation elements. 10.The apparatus according to claim 1, wherein the first transformation isa first matrix transformation and the second transformation is a secondmatrix transformation.
 11. The apparatus according to claim 10, whereinthe second matrix transformation is based on the following matrix:$\begin{bmatrix}1 & 1 & 1 & 1 \\1 & {- 1} & 1 & {- 1} \\1 & 1 & {- 1} & {- 1} \\1 & {- 1} & {- 1} & 1\end{bmatrix}.$
 12. The apparatus according to claim 10, wherein thefirst matrix transformation and the second matrix transformation aremerged to form a third matrix, wherein the third matrix is used in theat least one matrix multiplication.
 13. The apparatus according to claim10, wherein the first matrix transformation is applied by multiplying bya first matrix and the second matrix transformation is applied bymultiplying by a second matrix.
 14. An apparatus comprising: at leastone processor; and non-transitory system memory having stored thereoncomputer-executable instructions which, when executed by the at leastone processor, cause the apparatus to perform the following: perform atleast one transformation operation involving a plurality of correlationelements to generate at least one residual element in a first pluralityof residual elements and at least one residual element in a secondplurality of residual elements, the plurality of correlation elementsbeing dependent on an extent of correlation between the first pluralityof residual elements and the second plurality of residual elements, thefirst plurality of residual elements being useable to reconstruct afirst part of a first representation of a signal at a relatively highlevel of quality using a first part of a second representation of thesignal at the relatively high level of quality, the second plurality ofresidual elements being useable to reconstruct a second, different partof the first representation of the signal using a second, different partof the second representation of the signal; derive the first part of thesecond representation from a first set of one or more signal elements ina representation of the signal at a relatively low level of quality;derive the second part of the second representation from a second,different set of one or more signal elements in the representation ofthe signal at the relatively low level of quality; and use at least thefirst plurality of residual elements, the second plurality of residualelements, the first part of the second representation and the secondpart of the second representation to generate output data, wherein theat least one transformation operation is performed following theplurality of correlation elements being decoded, wherein the at leastone transformation operation implements at least one matrixmultiplication, and wherein the at least one matrix multiplicationimplements a first transformation operation applied to the plurality ofcorrelation elements and a second transformation operation applied to aresult of the first transformation.
 15. The apparatus according to claim14, wherein the first transformation operation and the secondtransformation both comprise inverse directional decompositions, theinverse directional decompositions being applied to decomposed elementsrepresenting each of average, horizontal, vertical and diagonaldirections.
 16. A method comprising: obtaining a first plurality ofresidual elements useable to reconstruct a first part of a firstrepresentation of a signal at a relatively high level of quality using afirst part of a second representation of the signal at the relativelyhigh level of quality, the first part of the second representation beingderivable from a first set of one or more signal elements in arepresentation of the signal at a relatively low level of quality;obtaining a second plurality of residual elements useable to reconstructa second, different part of the first representation of the signal usinga second, different part of the second representation of the signal, thesecond part of the second representation being derivable from a second,different set of one or more signal elements in the representation ofthe signal at the relatively low level of quality; and performing atleast one transformation operation involving at least one residualelement in the first plurality of residual elements and at least oneresidual element in the second plurality of residual elements togenerate a plurality of correlation elements, the plurality ofcorrelation elements being dependent on an extent of correlation betweenthe first plurality of residual elements and the second plurality ofresidual elements, wherein the at least one transformation operation isperformed prior to the plurality of correlation elements being encoded,wherein the at least one transformation operation implements at leastone matrix multiplication, and wherein the at least one matrixmultiplication implements a first transformation operation applied tothe first plurality of residual elements and the second plurality ofresidual elements and a second transformation operation applied to aresult of the first transformation.
 17. The method of claim 16, whereinthe first transformation operation and the second transformation bothcomprise directional decompositions, the directional decompositionsgenerating decomposed elements representing each of average, horizontal,vertical and diagonal directions.
 18. A method comprising: performing atleast one transformation operation involving a plurality of correlationelements to generate at least one residual element in a first pluralityof residual elements and at least one residual element in a secondplurality of residual elements, the plurality of correlation elementsbeing dependent on an extent of correlation between the first pluralityof residual elements and the second plurality of residual elements, thefirst plurality of residual elements being useable to reconstruct afirst part of a first representation of a signal at a relatively highlevel of quality using a first part of a second representation of thesignal at the relatively high level of quality, the second plurality ofresidual elements being useable to reconstruct a second, different partof the first representation of the signal using a second, different partof the second representation of the signal; deriving the first part ofthe second representation from a first set of one or more signalelements in a representation of the signal at a relatively low level ofquality; deriving the second part of the second representation from asecond, different set of one or more signal elements in therepresentation of the signal at the relatively low level of quality; andusing at least the first plurality of residual elements, the secondplurality of residual elements, the first part of the secondrepresentation and the second part of the second representation togenerate output data, wherein the at least one transformation operationis performed following the plurality of correlation elements beingdecoded, wherein the at least one transformation operation implements atleast one matrix multiplication, and wherein the at least one matrixmultiplication implements a first transformation operation applied tothe plurality of correlation elements and a second transformationoperation applied to a result of the first transformation.
 19. Themethod of claim 18, wherein the first transformation operation and thesecond transformation both comprise inverse directional decompositions,the inverse directional decompositions being applied to decomposedelements representing each of average, horizontal, vertical and diagonaldirections.
 20. The method of claim 18, wherein the first transformationis a first matrix transformation and the second transformation is asecond matrix transformation, wherein the second matrix transformationis based on the following matrix: $\begin{bmatrix}1 & 1 & 1 & 1 \\1 & {- 1} & 1 & {- 1} \\1 & 1 & {- 1} & {- 1} \\1 & {- 1} & {- 1} & 1\end{bmatrix}.$ and, wherein the first matrix transformation and thesecond matrix transformation are merged to form a third matrix, whereinthe third matrix is used in the at least one matrix multiplication.